TrackIQ Comparison Report

Generated at 2026-02-22T03:33:34.068050+00:00

Executive Summary

Overall winner: MiniCluster. MiniCluster won 1 of 5 comparable metrics. Largest deltas: throughput_samples_per_sec (+inf%), latency_p50_ms (+0.00%), latency_p95_ms (+0.00%). No regressions exceeded 5.0%. Consistency findings: none.

latency_p50_ms: +0.00%latency_p95_ms: +0.00%

How To Read This Report

Comparable metrics: 5Missing/non-comparable: 9Flagged regressions: 0Consistency findings: 0

AutoPerfPyResult A

Tool: autoperfpy 1.0
Platform: unknown (Windows 11)
Framework: pytorch 2.5.1+cu121
Workload: html_report_data (inference)
Timestamp: 2026-02-22T02:22:55.013502+00:00

MiniClusterResult B

Tool: minicluster 0.1.0
Platform: CPU (Windows 11)
Framework: pytorch 2.5.1+cu121
Workload: distributed_training_validation (training)
Timestamp: 2026-02-21T04:05:37.557945

Metric Comparison

Metric AutoPerfPy MiniCluster Abs Delta % Delta Winner
communication_overhead_percentN/AN/AN/AN/AN/A
decode_tpt_msN/AN/AN/AN/AN/A
energy_per_step_joulesN/A3.9279N/AN/AN/A
latency_p50_ms0.00000.00000.0000+0.00%tie
latency_p95_ms0.00000.00000.0000+0.00%tie
latency_p99_ms0.00000.00000.0000+0.00%tie
memory_utilization_percent0.00000.00000.0000+0.00%tie
performance_per_wattN/A8.1625N/AN/AN/A
power_consumption_wattsN/A24.8406N/AN/AN/A
scaling_efficiency_pctN/AN/AN/AN/AN/A
temperature_celsiusN/A36.6756N/AN/AN/A
throughput_samples_per_sec0.0000202.7609202.7609infMiniCluster
tokens_per_sec22.0401N/AN/AN/AN/A
ttft_msN/AN/AN/AN/AN/A

Visual Overview

Consolidated graph views for quick comparison validation.

Top Normalized Deltas

Positive favors MiniCluster; negative favors AutoPerfPy.

Metric Family Deltas

Family-level signed summary of normalized deltas.

Winner Distribution

Metric-level outcome share across comparable metrics.

Confidence Distribution

Data-strength breakdown for metric conclusions.

Normalized Metric Deltas

Positive values indicate an advantage for MiniCluster; negative values favor AutoPerfPy.

Metric Family Direction Raw Delta % Normalized Delta % Visual Advantage
latency_p50_mslatencylow+0.00%-0.00%
tie
latency_p95_mslatencylow+0.00%-0.00%
tie
latency_p99_mslatencylow+0.00%-0.00%
tie
memory_utilization_percentmemorycontext+0.00%+0.00%
context

Metric Family Delta Waterfall

Family-level mean of normalized metric deltas (context-only metrics excluded).

Family Metrics Normalized Delta % Visual Winner
latency3+0.00%
tie

Metric Availability Confidence Matrix

Confidence is based on metric availability in both results.

Metric Family AutoPerfPy Available MiniCluster Available Direction Confidence
communication_overhead_percentcommunicationnonolownone
decode_tpt_msothernonolownone
energy_per_step_joulesefficiencynoyeslowinsufficient
latency_p50_mslatencyyesyeslowstrong
latency_p95_mslatencyyesyeslowstrong
latency_p99_mslatencyyesyeslowstrong
memory_utilization_percentmemoryyesyescontextstrong
performance_per_wattperformancenoyeshighinsufficient
power_consumption_wattsefficiencynoyeslowinsufficient
scaling_efficiency_pctothernonohighnone
temperature_celsiusothernoyeshighinsufficient
throughput_samples_per_secperformanceyesyeshighstrong
tokens_per_secotheryesnohighinsufficient
ttft_msothernonolownone

Consistency Analysis

No consistency regressions detected or all-reduce step data was unavailable.

Platform Comparison